Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cluster provision functionality #2452

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

tillrohrmann
Copy link
Contributor

@tillrohrmann tillrohrmann commented Dec 20, 2024

After starting the metdata store service and the grpc server, the node will try to initialize itself by joining an existing cluster or provisioning a new cluster. Either of the two actions will return the NodeId of the node. Moreover, the initialization procedure makes sure that the latest metadata is fetched from the metadata store.

The cluster provisioning will ensure that all relevant metadata has been written to the metadata store. This includes the initial NodesConfiguration, a PartitionTable that includes the number of partitions and an empty Logs that contains the default log provider configuration.

To use the cluster provisioning feature, one needs to disable allow-bootstrap. Otherwise the cluster will try to auto provision itself. You can use restatectl to provision the cluster via:

restatectl cluster provision --address http://localhost:5122 --num-partitions 42 --bifrost-provider replicated --replication-property 13

This fixes #2409.

crates/core/protobuf/node_ctl_svc.proto Outdated Show resolved Hide resolved
crates/core/protobuf/node_ctl_svc.proto Outdated Show resolved Hide resolved
crates/core/protobuf/node_ctl_svc.proto Outdated Show resolved Hide resolved
crates/core/protobuf/node_ctl_svc.proto Outdated Show resolved Hide resolved
crates/local-cluster-runner/src/node/mod.rs Show resolved Hide resolved
crates/node/src/init.rs Outdated Show resolved Hide resolved
crates/node/src/init.rs Outdated Show resolved Hide resolved
crates/node/src/init.rs Outdated Show resolved Hide resolved
crates/node/src/init.rs Outdated Show resolved Hide resolved
crates/types/src/node_id.rs Show resolved Hide resolved
After starting the metdata store service and the grpc server, the node will
try to initialize itself by joining an existing cluster. Additionally each node
exposes an provision cluster grpc call with which it is possible to provision
a cluster (writing the initial NodesConfiguration, PartitionTable and Logs).
Nodes can only join after the cluster is provisioned.

This fixes restatedev#2409.
@tillrohrmann
Copy link
Contributor Author

I've updated the PR wrt your feedback @AhmedSoliman and it turned out to remove quite a bit of the plumbing I initially introduced :-) The grpc handler contains the logic to provision the required metadata for a cluster. Whether a metadata store requires an explicit provision step or not, is now encapsulated in the MetadataStore implementation (the trait now exposes a provision method). This also removed the previous requirement that an embedded metadata store needed to be co-located with the provisioning node.

Joining a cluster happens independently of the provision step. This means that the node which executes the cluster provision will bump its own NodeId immediately when joining the cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add provision command to bootstrap metadata store and write initial NodesConfiguration
2 participants